The invention relates generally to software testing. More specifically, the invention relates to methods and systems that dynamically reprioritize test cases used in Model-Based Testing (MBT) during test execution.
Model-Based Testing (MBT) is the application of Model-Based Design (MBD) for designing and executing the necessary test artifacts such as test cases and test scripts to perform software testing. A test model is used that describes all aspects of test data needed for testing such as test inputs, expected outputs and test execution environment. The test model is typically derived from a design model that describes aspects of a System Under Test (SUT). A wide range of test generation methods have been developed and are currently provided by many test generation tools supporting the MBT approach.
The number of test cases generated depends on two factors: 1) the method to be selected, and 2) the underlying design model. The design model is an architectural artifact that describes the detailed design of a system using diagrams such as workflow diagrams. While the first factor is selected by a tester, the second factor can not be influenced by the tester. The selected method is a test case generation algorithm that depends on different criteria on how to derive test cases. For example, the criteria may cover all test input variations within the design model or cover all activities within the design model. Accordingly, the test case generation algorithm generates test cases that either fulfills the test input or the activity coverage criteria. Since time allocated for testing is limited, only a finite number of test cases may be executed.
The selection of the right method is important to achieve an appropriate coverage of the underlying architecture to fulfill a certain quality attribute. Therefore, the order in which the test cases are executed is very important to reveal as many defects as possible in the given period of time. The underlying architecture is how the codes or methods and how the features or components are structured for a system test.
The objective of a test case is to fail during execution and thereby indicate that a potential illegal behavior of the system was uncovered. However, test cases that fail during a test run that do not result from different causes may reveal the same defect. Within a test run a collection of test cases is planned to be executed against a specific version of the SUT.
Current test case prioritization approaches do not consider the dependency between test cases during execution. Most approaches prioritize regression test cases where the objective is to retest a system or component to verify that modifications previously performed have not caused unintended effects. These approaches do not provide dynamic reprioritization during test case execution.
What is desired is a test case prioritization method that reorders test case execution towards their potential to cover divergent defects within a system dynamically during a test run.